An Evaluation Framework for Spoken Language Processing

In our paper we describe the structure and implementation of a semi-automatic evaluation environment for natural language processing in speech understanding systems. The major goal of this evaluation framework is to support the construction of linguistic processors and their accompanying linguistic knowledge bases (grammars) in order to reduce development costs and raise quality. Our evaluation framework allows for giving feedback on eeects of grammar extensions or changes to the grammar writer in terms of absolute gures, i.e. diierent versions of a grammar may be compared through evaluation. This also gives opportunity to make a cost and beneets analysis, as diier-ent parsing strategies may be compared directly. As a result, our evaluation framework can guide the development of a linguistic processor and serve as well as a means to compare diierent linguistic processors by computing comparable characteristic gures.